SPECIFICATION 

Electronic Version 1.2.8 
Stylesheet Version 1 .0 

Sum-of-Absolute-Difference 
Checking of Macroblock Borders 
for Error Detection in a Corrupted 
MPEG-4 Bitstream 

Background of Invention 

[0001] This invention relates to video compression, and more particularly to error 
recovery from macroblock location errors. 

[0002] Multimedia-rich personal computers (PCs) and various other computing devices 
have delivered video feeds to users over the Internet. Processing of video bitstreams 
or feeds is among the most data-intensive of all common computing applications. 
Limited communication-line bandwidth limits the quality of Internet video, which is 
often delivered in small on-screen windows with jerky movement. 

[0003] To mitigate the problems of large video streams, various video-compression 

techniques have been deployed. Compression standards, such as those developed by 
the motion-picture-experts group (MPEG), have been widely adopted. These 
compression techniques are lossy techniques, since some of the picture information is 
discarded to increase the compression ratio. However, compression ratios of 99% or 
more have been achieved with minimal noticeable picture degradation. 

[0004] Portable hand-held devices such as personal-digital-assistants and cellular 

telephones are widely seen today. Wireless services allow these devices to access data 
networks and even view portions of web pages. Currently the limited bandwidthuof 
these wireless networks limits the web viewing experience to mostly text-based 
portions of web pages. However, future wireless networks are being planned that 
should have much higher data transmission rates, allowing graphics and even video to 
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be transmitted to portable computing and communication devices. 

[0005] Although proponents of these next-generation wireless networks believe that 

bandwidths will be high enough for high-quality video streams, the actual data rates 
delivered by wireless networks can be significantly lower than theoretical maximum 
rates, and can vary with conditions and local interference. Due to its high data 
requirements, video is likely to be the most sensitive service to any reduced data 
rates. Interference can cause intermittent dropped data over the wireless networks. 

[0006] Next-generation compression standards have been developed for transmitting 
video over such wireless networks. The MPEG-4 standard provides a more robust 
compression technique for transmission over wireless networks. Recovery can occur 
when parts of the MPEG-4 bitstream is corrupted. 

[0007] Figure 1 shows that an image frame is divided into rows and columns of 

macroblocks. The MPEG standard uses a divide-and-conquer technique in which the 
video sequence is divided into individual image frames known as video object planes 
(VOPs), and each frame is divided into rows and columns of macroblocks. Each 
macroblock is a rectangle of 1 6 by 1 6 pixels. 

[0008] Various window sizes and image resolutions can be supported by MPEG standards. 
For example, one common format is an image frame of 1 76 by 1 44 pixels. The image 
frame is divided into 9 rows of macroblocks, with each row having 1 1 macroblocks. A 
total of 99 macroblocks are contained in each frame. 

[0009] The macroblocks are arranged in a predetermined order, starting in the upper left 
with the first macroblock (MB #1). The second macroblock, MB#2, is to the right of 
MB#1 in the first row, followed by macroblocks #3 to MB#1 1 in the first row. The 
second row contains MB#12 to MB#22. The last row contains MB#89 to MB#99. Of 
course, other image sizes and formats can have the macroblocks in rows of various 
lengths, and various numbers of rows. 

[0010] When an image frame is encoded, each macroblock is encoded in order, starting 
with MB#1 in the first row, and continuing with MB#2 to MB#1 1 in the first row, then 
MB#12 to MB#22 in the second row, and on until the last row with MB#89 to MB#99. 
The macroblocks are arranged in the bitstream into one or more video packets (VP). 
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Each video packet contains a header, allowing for some error recovery to occur. For 
example, the number of the next macroblock in the frame, which is the first 
macroblock in the new video packet, is included in the VP header. 

[001 1] Figure 2 shows a MPEG-4 bitstream that is composed of video object planes and 

video packets. The video is sent as a series of picture frames known as video object 
planes (VOP). These picture frames are replaced at a fixed rate, such as every 30 
milliseconds to give the illusion of picture movement. Rather than transmit every pixel 
on each line, the picture is divided into macroblocks and compressed by searching for 
similar macroblocks in earlier or later frames. The macroblock can then be replaced 
with a motion vector or pixel changes. 

[001 2] Video object planes VOP 1 0, 1 2 are two frames in a sequence of many frames that 
form a video stream. Pixel data in these planes are compressed using macroblock- 
compression techniques that are well-known and defined by the MPEG-4 standard. 
The compressed picture data is divided into several video packets (VP) for each video 
object plane VOP. 

[001 3] Each video object plane begins with a VOP start code, such as VOP start code 20 

which begins VOP #1 (10), an VOP start code 21, which begins VOP #2 (12). First video 
object plane VOP 10 has VOP header 22 that follows VOP start code 20, and data field 
24 which contains the beginning of the picture data for VOP 1 0. After a predetermined 
number of macroblocks or amount of data, such as 1 00 to 1 000 bits, a new video 
packet begins with resync marker 30 and VP header 32. Data field 34 continues with 
the picture data for several more macroblocks of VOP 10. Other video packets follow, 
each beginning with a resync marker and VP header, followed by a data field with 
more macroblocks of picture data for VOP 1 0. The last video packet VP #N in VOP 1 0 
begins with resync marker 31 and VP header 33, and is followed by the final 
macroblocks for VOP 1 0, in data field 35. 

[0014] The second video object plane VOP 12 begins with VOP start code 21 and VOP 

header 23, and is followed by data field 25, which has the first macroblocks of picture 
data for the second picture frame, VOP 1 2. Other video packets follow for VOP 1 2 with 
the other macroblocks. 
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[001 5] The VOP headers include a VOP coding type (I for intra-coded, no prediction, P for 
prediction from Previous VOP, B for Bi-directional prediction from previous and next 
VOPs), VOP time, rounding type, quantization scale, f_code, while the VP headers 
include a macroblock number for the first macroblock in the packet, quantization 
scale, VOP coding type and time. The headers can include other information as well. 

[001 6] The VOP start codes and VP resync markers contain unique bit patterns that do 

not occur in the headers or data fields. The start code begins with a string of 23 zero 
bits. The picture data in the macroblock data fields are encoded so that they never 
have such a long string or run of zero bits. Likewise, the headers do not have such a 
long run of zero bits. Thus the start code is unique within the video bitstream, 
allowing a bitstream decoder to easily detect the start code. 

[001 7] Figure 3 is a flowchart for macroblock counting and decoding. For each VOP or 
frame, the MPEG decoder decodes one macroblock after another in order. The 
macroblocks do not themselves contain a macroblock number or identifier. Instead, a 
macroblock counter in the decoder is incremented for each macroblock processed. 
The macroblock counter thus keeps track of which macroblock is being decoded. 

[001 8] Each video packet header also contains a macroblock number for the next 

macroblock in the new packet. This header macroblock number can be compared to 
the macroblock counter as a check. The first video packet in a frame has a VOP header 
rather than a VP header. The macroblock counter is reset for each new VOP, so a 
macroblock number is not needed for the first packet of each frame. 

[001 9] A new video packet N is detected when a resync marker is found in the bitstream, 
step 72. The VP header that follows the resync marker is decoded, step 74. This 
header contains the header macroblock number, which is the number for the first 
macroblock in the new video packet N. When the video packet is the first video packet 
in a frame, the header macroblock number is implied to be one, the same value as the 
reset or initialized macroblock counter. 



[0020] 



The decoder compares the header macroblock number that was just read from the 
VP header to the macroblock counter value, step 76. Most of the time, the values 
match. Decoding of all the macroblocks in the video packet can proceed, step 80. The 
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macroblock counter is incremented for each macroblock processed. 

[0021] When an error is detected when decoding the macroblocks in video packet N, step 
82, then the decoder may conceal the error, step 84. Errors can be concealed by using 
pixels from a previous frame rather than the pixels, error terms, or motion vectors 
encoded with the macroblocks in the video packet. 

[0022] When no decoding errors are detected, and all the macroblocks in packet N have 
been processed, then decoding can continue with the next video packet, step 72. 

[0023] Occasionally, the header macroblock number does not match the macroblock 

counter, step 76. Some kind of error has occurred. When the previous video packet N- 
1 had a decoding error, step 78, then the error may have caused the macroblock 
counter to get off count. The header macroblock number is assumed to be correct and 
the macroblock counter wrong. The header macroblock number in the new video 
packet N is used to over-write or update the macroblock counter. The header 
macroblock number is thus loaded into the macroblock counter, step 88. The 
macroblock counter is then incremented for each macroblock decoded in the current 
video packet N, step 80. 

[0024] When the previous packet N-l did not have a detected decoding error, step 78, 
then the macroblock counter is probably correct. Perhaps the error is in the new VP 
header, causing the header macroblock number to be read incorrectly. The header 
macroblock number is ignored or discarded, while the macroblock counter is used 
without any update, step 79. The macroblock counter is then incremented for each 
macroblock decoded in the current video packet N, step 80. 

[0025] Another error may be detected when decoding the macroblocks in video packet N, 
step 82. The decoder may conceal the error, step 84, by using pixels from a previous 
frame rather than the pixels, error terms, or motion vectors encoded with the 
macroblocks in the current video packet N. 

[0026] Figure 4A shows recovery from a bit error in the header macroblock number. 

When the bitstream is transmitted over a wireless network, some corruption of the 
data is possible. In this example, a bit error occurs in the bitstream in VP header 33, 
causing an incorrect header macroblock number to be read. 
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[0027] The VOP frame begins with start code 20 and VOP header 22. The macroblock 

counter is reset to 1 (or 0 if the first MB is considered to be MB#0) at the start of each 
new VOP. The macroblock counter is incremented for each macroblock in data field 
24. The counter can be pre- or post-incremented with each macroblock, depending 
on the exact counter timing used and any pipelining. As the last macroblock (MB#52) 
in data field 24 is processed, the macroblock counter reads 52. The macroblock 
counter is then incremented to 53 before the next macroblock (MB#53) in data field 
34 is processed. 

[0028] The next resync marker 30 begins video packet #2. VP header 32 following resync 
marker 30 is decoded. VP header 32 contains the header macroblock number 53, 
since the first macroblock in this packet is MB#53. Since the macroblock counter and 
the header macroblock number match, no error is detected and data processing 
resumes with data field 34 in the second video packet. The macroblock counter is 
incremented for each macroblock in data field 34, from MB#53 to MB#80. 

[0029] Resync marker 31 is detected for the third video packet. VP header 33 is decoded, 

and the new header macroblock number is read and compared to the macroblock 
counter value. The macroblock counter is pre-incremented from 80 to 81 , ready for 
MB#81 as the next macroblock. 

[0030] However, a bit error occurs in VP header 33. The header macroblock number is 
corrupted so that the wrong value is read. Although the MPEG encoder wrote the 
correct value 81 to VP header 33, the bit error caused a different number, such as 33 
or 75, to be read. This wrong header macroblock number does not match the 
macroblock counter value of 81 . 

[0031] Since no error was detected in the previous video packet (#2), the macroblock 
counter is assumed to be correct and the header macroblock number is assumed to 
be wrong. The header macroblock number is ignored and the macroblock counter is 
used for identifying the macroblocks in data field 35. Data processing continues with 
macroblocks MB#81 to MB#99 in data field 2 5. The macroblock counter is 
incremented for each macroblock in data field 25. 

[0032] 

Correctly guessing that the header macroblock number was wrong reduces the 
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amount of lost data when a bitstream error occurs. Decoding of macroblocks can 
proceed without any error when the only error was the header macroblock number. 

[0033] Unfortunately, an incorrect guess for other kinds of bit errors is more difficult to 
recover from. Figure 4B shows an undetected bit error in the macroblock data that 
corrupts the macroblock counter. Although start code 20 and VOP header 22 for VOP 
#1 are detected and decoded, a bit error in macroblock data in data field 24 occurs. 

[0034] Some of the early macroblocks MB#1 , MB#2 ... in data field 24 are properly 

decoded, but an error occurs before the last macroblock MB#52 in data field 24 is 
processed. This error is not detected, but the start of some of the macroblocks in data 
field 24 is not recognized. The decoder combines several macroblocks of data 
together into a single corrupted macroblock. 

[0035] Since some of the macroblocks in data field 24 are not recognized, the 

macroblock counter is not incremented enough times. The macroblock counter is only 
incremented for recognized macroblocks. In this example, the macroblock counter is 
off by 5, reading 47 for the last macroblock MB#52 in data field 24. 

[0036] Resync marker 30 for the second video packet (VP) is detected, and VP header 32 
is decoded. A correct value of 53 is read from VP header 32 as the header macroblock 
number. 

[0037] The macroblock counter is pre-incremented from 47 to 48, but does not match 
the header macroblock number of 5 3. An error is detected. However, since the bit 
error in data field 24 was not detected, the macroblock counter is assumed to be 
correct and the header macroblock number is assumed to be wrong and is ignored. 
This is a decoder mistake. 

[0038] Macroblocks MB#53 to MB#80 in data field 34 are decoded, but as interpreted as 
macroblocks MB#48 to MB#75. The macroblock counter is incremented for each 
macroblock, but starts out the video packet with the incorrect value of 48. 

[0039] 

Resync marker 31 is detected, and VP #3 header 33 is decoded. The header 
macroblock number of 81 is read from VP header 33, but the macroblock counter 
reads 76 once it is pre-incremented. Since no error was detected in the previous video 
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packet #2, the decoder assumes that the macroblock counter is correct and the 
header macroblock number is wrong. This is an incorrect assumption by the decoder. 

[0040] Macroblock MB#81 is interpreted by the decoder as MB#76, since the macroblock 
counter reads 76. Macroblocks MB#82 to MB#99 are interpreted incorrectly as 
macroblocks #77 to #94. The decoder finally recovers when the macroblock counter is 
reset the by start code for the next VOP #2. 

[0041] What is desired is a bitstream decoder that more accurately responds to mis- 
matches in the macroblock number. A robust decoder is desired that can more quickly 
recover from bitstream errors. An MPEG-4 decoder that can recover from a corrupted 
bitstream and mis-matching macroblock number is desirable to minimize loss of 
picture data. An MPEG-4 decoder that can more accurately choose either the header 
macroblock number or the macroblock counter value when a mismatch occurs is 
desired. 

Brief Description of Drawings 

[0042] Figure 1 shows that an image frame is divided into rows and columns of 
macroblocks. 

[0043] Figure 2 shows a MPEG-4 bitstream that is composed of video object planes and 

video packets. 

[0044] Figure 3 is a flowchart for macroblock counting and decoding. 

[0045] Figure 4A shows recovery from a bit error in the header macroblock number. 

[0046] Figure 4B shows an undetected bit error in the macroblock data that corrupts the 
macroblock counter. 

[0047] Figure 5A shows a grid of macroblock boundaries imposed over image shapes 
when no error occurs. 

[0048] Figure 5B shows that image shapes are cut by macroblock boundaries when a 
macroblock sequencing error occurs. 

[0049] Figures 6A.B highlight checking for image errors along macroblock boundaries. 
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[0050] Figure 7 is a flowchart for error detection by macroblock-border sum-of- 

absolute-difference threshold comparison. 

[005 1 ] Figure 8 is a block diagram of an MPEG decoder with error detection using 

macroblock-border SAD checking. 

[0052] Figure 9 shows border- SAD detection of a bit error in the macroblock data that 

corrupts the macroblock counter. 

Detailed Description 

[0053] The present invention relates to an improvement in macroblock tracking. The 
following description is presented to enable one of ordinary skill in the art to make 
and use the invention as provided in the context of a particular application and its 
requirements. Various modifications to the preferred embodiment will be apparent to 
those with skill in the art, and the general principles defined herein may be applied to 
other embodiments. Therefore, the present invention is not intended to be limited to 
the particular embodiments shown and described, but is to be accorded the widest 
scope consistent with the principles and novel features herein disclosed. 

[0054] The inventor realizes that better error recovery from macroblock sequencing 

errors could be accomplished if it is known whether the macroblock counter or the 
header macroblock number is wrong when a mismatch occurs. Although it may not be 
know for certainty which value is in error, the inventor realizes that the image itself 
can be used to more accurately decide which macroblock number is in error. 

[0055] The inventor further recognizes that the macroblock boundaries are arbitrary 

divisions of the image itself. These macroblock boundaries frequently cross through 
image shapes. When macroblocks are arranged in proper order, the macroblock 
boundaries are unlikely to also be boundaries of image shapes. Sharp discontinuities 
at macroblock boundaries are likely to indicate a decoding error. The inventor checks 
macroblock boundaries for such sharp discontinuity to further detect when a 
macroblock decoding error has occurred. 

[0056] 

Figure 5A shows a grid of macroblock boundaries imposed over image shapes 
when no error occurs. Three image shapes are displayed to a user in this example. A 
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rhombus occupies parts of four macroblocks MB#2,3,7,8 while a triangle straddles the 
intersection of four macroblocks MB#4,6,9,10. A rectangle is located along the 
boundaries of six macroblocks MB#7,8,9,1 2,1 3,14. 

[0057] Since the macroblocks have 1 6 rows and 1 6 columns pixels, and only 2 rows and 
2 columns are on the macroblock edge, the probability that an edge of an image 
shape falls exactly along a macroblock edge is only 2 in 1 6, or about 1 2%. Of course, 
image shapes are not necessarily parallel to the macroblock edges. In this simplified 
example, the image frame has only 1 5 macroblocks. Typical frames have many more 
rows and columns of macroblocks. 

[0058] Figure 5B shows that image shapes are cut by macroblock boundaries when a 

macroblock sequencing error occurs. In this example, a macroblock sequencing error 
occurs. Macroblock MB#4 is lost or skipped during decoding, so that macroblock 
MB#5 is interpreted as MB#4. The macroblock counter is off by one, so that each 
macroblock after MB#3 is shifted left by one macroblock position, or back to the end 
of the previous row. For example, MB#6 should be the left-most macroblock of the 
second row, but instead is shifted to the last macroblock of the first row, where MB#5 
should be located. 

[0059] Image shapes that straddle macroblock boundaries are sliced and shifted when 
such macroblock sequencing errors occur. For example, the top of the rhombus is 
correctly located in MB#2,3, but the bottom half of the rhombus is shifted to the left 
along with MB#7,8. The result is a new bisecting edge of the rhombus along the 
bottom of macroblocks MB#2,3, and the top of macroblocks MB#7,8. 

[0060] The data of MB#4 is lost, so a portion of the triangle is missing. This also results 
in two new edges of the triangle image shape that fall along macroblock boundaries - 
one new edge between MB#3 and MB#5, and another new edge between MB#3 and 
MB#9. 



[0061] 



When the macroblock sequencing error is not detected, image shapes that are 
encoded entirely by macroblocks after the sequencing error are shifted in the frame, 
but do not have new edges. Thus the rectangle is shifted to the left by one column of 
macroblocks, but does not have any new bisecting edges, as do the rhombus and 
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triangle. 

[0062] When macroblocks are damaged and the error is detected, the damaged pixels can 
be hidden or concealed. New image shape edges can still appear along macroblock 
boundaries, although part of the image shapes may not be displayed. 

[0063] The inventor detects these new bisecting edges or discontinuities that are caused 
by macroblock sequencing errors. A sum-of-the-absolute difference (SAD) technique 
is use to quantify such discontinuities and make a decision on whether such a 
macroblock-counter sequencing error is present. 

[0064] Figures 6A,B highlight checking for image errors along macroblock boundaries. 

Figure 6A highlights checking for a row-to-row discontinuities along macroblock row 
boundaries. A current macroblock MB#1 3 is being decoded. The upper edge of MB#1 3 
meets the lower edge of MB#2 above it. Current MB#1 3 contains an upper row of 
sixteen pixels Al , A2, A3 ... Al 6, while previous-row MB#2 has a lower row of sixteen 
pixels Bl, B2, B3, ... B16. 

[0065] When no errors are present, it is unlikely that a sharp discontinuity occurs along 
the macroblock edge. Thus pixel Al is close in value to pixel Bl, pixel A2 is close in 
value to pixel B2, etc. However, when a sharp discontinuity occurs, pixel Al has a 
different color value than adjacent pixel B2 in another macroblock. Likewise, pixel A2 
has a different value than pixel B2. 

[0066] The differences in pixel values along the edge can be quantified using the sum- 
of-the-absolute difference (SAD). The difference in pixel values of Al and Bl are 
generated, and the sign bit dropped to get the absolute difference of pixels Al and 
Bl . The absolute difference of the other 1 5 pixel-pairs, such as A2, B2, and Al 6, Bl 6, 
are also generated. Finally all 1 6 absolute differences are summed, to obtain the SAD 
for this edge. 

[0067] When a sharp discontinuity occurs along this macroblock boundary, the SAD is 
large. When no discontinuity or image edge occurs along the macroblock boundary, 
the SAD is small or even zero. Thus the SAD can be used to quantify the degree of 
discontinuity. 
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[0068] Figure 6B highlights checking for discontinuities between adjacent macroblocks in 

a row. Current macroblock MB#1 3 is being decoded. The left edge of MB#1 3 meets 
the right edge of MB#1 2 next to it. Current MB#1 3 contains a leftmost column of 
sixteen pixels CI , C2, C3 ... CI 6, while previous-row MB#2 has a rightmost column of 
sixteen pixels Dl , D2, D3, ... D16. 

[0069] When no error occurs, it is unlikely that an edge of an image shape falls exactly 
along the edges of macroblock MB#1 2, 1 3. However, a sharp discontinuity along the 
macroblock edge suggest that a macroblock counter error has occurred. 

[0070] The degree of difference in the pixels CI , C2 ... CI 6 and Dl , D2, ... D6 can be 
quantified using the sum-of-the-absolute difference (SAD). The SAD for both the 
upper and left edges of current macroblock MB#1 3 are calculated as: 

[0071] 

j-i i-i 

[0072] The current macroblock is incremented along with the macroblock counter as the 
macroblocks in the video packet are decoded and their SAD's generated. A maximum 
SAD can be kept as the macroblocks in the video packet are decoded and their SAD's 
generated. The SAD's can be generated on the fly as each packet is being decoded, or 
the error controller can go back and generate the SAD's for the previous packet once a 
macroblock number mismatch is detected in a new packet. 

[0073] Figure 7 is a flowchart for error detection by macroblock-border sum-of- 

absolute-difference threshold comparison. For each VOP or frame, the MPEG decoder 
decodes one macroblock after another in order. The macroblocks do not themselves 
contain a macroblock number or identifier. Instead, the macroblock counter in the 
decoder is incremented for each macroblock processed. 

[0074] 

A new video packet N is detected when a resync marker is found in the bitstream, 
step 72. The VP header that follows the resync marker is decoded, step 74. This 
header contains the header macroblock number, which is the number for the first 
macroblock in the new video packet N. When the video packet is the first video packet 
in a frame, the header macroblock number is implied to be one, the same value as the 
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reset or initialized macroblock counter. 

[0075] The decoder compares the header macroblock number that was just read from the 
VP header to the macroblock counter value, step 76. Most of the time, the values 
match. Decoding of all the macroblocks in the video packet can proceed, step 80. The 
macroblock counter is incremented for each macroblock processed. 

[0076] When an error is detected when decoding the macroblocks in video packet N, step 
82, then the decoder may conceal the error, step 84. Errors can be concealed by using 
pixels from a previous frame rather than the pixels, error terms, or motion vectors 
encoded with the macroblocks in the video packet. Averaging of surrounding 
macroblock pixels can also be used to conceal errors. When no decoding errors are 
detected, and all the macroblocks in packet N have been processed, then decoding 
can continue with the next video packet, step 72. 

[0077] Occasionally, the header macroblock number does not match the macroblock 

counter, step 76. Some kind of error has occurred. When the previous video packet N- 
1 had a decoding error, step 78, then the error is likely to have caused the macroblock 
counter to get off count. The header macroblock number is assumed to be correct and 
the macroblock counter wrong. The header macroblock number in the new video 
packet N is used to over-write or update the macroblock counter. The header 
macroblock number is thus loaded into the macroblock counter, step 88. The 
macroblock counter is then incremented for each macroblock decoded in the current 
video packet N, step 80. 

[0078] SAD Used to Decide Whether Header or Macroblock Counter is Wrong 

[0079] When the macroblock number from the header and the macroblock counter do not 
match, and the previous packet N-l did not have a detected decoding error, step 78, 
then it is not certain whether the macroblock counter or the header is correct. Either 
the macroblock counter or the header must be in error, since there values do not 
match (determined earlier in step 76). 

[0080] j ne i S used to verify the previous video packet. The SAD of each macroblock 
in the last packet N-l is calculated, step 86. Macroblock data from earlier packets 
such as packet N-2 may need to be fetched from a pixel memory since the row above 



App_ID- 10065404 



Page 13 of 37 



the macroblocks in packet N-1 may be from an earlier packet. When the current 
macroblock is in the first row of macroblocks, then only the left-edge SAD and not the 
upper SAD is calculated. Likewise, when the current macroblock is in the first column 
of macroblocks in the frame, then the left-edge SAD is skipped. 

[0081] As the SAD for each macroblock in the previous packet N-1 is generated, it's SAD 
can be accumulated as a running SAD for the packet. Once the last macroblock in the 
video packet has its SAD generated and accumulated, the accumulated SAD is 
compared to a threshold, step 90. This threshold value can be set based on 
experience with similar images, or can be a somewhat arbitrary value. In one 
embodiment, the threshold is 32 x 35, or 1 12. The threshold is set high enough so 
that error-free images do not have SAD's above the threshold, but the SAD for video 
packets with macroblock counting errors are above the threshold. 

[0082] When the maximum SAD is above the threshold, step 90, then there are many 
discontinuities, more than typical. It is likely that the macroblock counter went off 
sequence in the last video packet N-1 . The macroblock counter is then re-loaded with 
the header macroblock number, step 88. The macroblocks in the new video packet N 
can then be decoded, step 80, using the updated macroblock counter value. 

[0083] When the SAD is below the th reshold, step 90, then the image has few 

discontinuities. It is likely that the correct image was generated when the macroblocks 
in the last video packet N-1 were decoded. The error is likely in the new VP header, 
causing the header macroblock number to be read incorrectly. The header macroblock 
number is ignored or discarded, while the macroblock counter is used without any 
update, step 79. The macroblock counter is then incremented for each macroblock 
decoded in the current video packet N, step 80. 

[0084] Another error may be detected when decoding the macroblocks in video packet N, 
step 82. The decoder may conceal the error, step 84, by using pixels from a previous 
frame rather than the pixels, error terms, or motion vectors encoded with the 
macroblocks in the current video packet N. 



[0085] 



Figure 8 is a block diagram of an MPEG decoder with error detection using 
macroblock-border SAD checking. Many other blocks can be present in an actual 
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MPEG decoder but are not shown for simplicity. Resync detector 50 detects the start of 
a new video packet and triggers header decoder 52 to decode the VP header. The 
header macroblock number is read and sent to comparator 70. 



header and before the next resync marker is detected. Macroblock counter 60 is 
incremented for each macroblock decoded, and is reset at the start of each new frame 
or VOP. Macroblock counter 60 may be reset to 1 rather than to 0. Macroblock counter 
60 keeps track of which macroblock is being decoded, and instructs pixel writer 62 to 
write the current macroblock's pixels in the proper location in memory 56 so that the 
macroblock's pixels appear in the proper location within the frame. Pixel writer 62 
may first read pixels from memory 56 from an earlier frame, then translate these 
pixels to a new location using a motion vector. Error terms may also be added to the 
pixels read. 

[0087] Macroblock counter 60 also sends the macroblock number to comparator 70. At 
the start of a new video packet, when VP header decoder 52 sends the header 
macroblock number to comparator 70, comparator 70 compares the macroblock 
numbers from the VP header and macroblock counter 60. At other times comparator 
70 is idle, such as when macroblocks are being decoded after the header. 

[0088] When comparator 70 determines that the headers do not match, error controller 
58 is activated. Decoding of new macroblocks by macroblock decoder 54 may be 
halted until the error is resolved. When an error was also detected in the previous 
video packet, error controller 5 8 assumes that the macroblock number in macroblock 
counter 60 is incorrect. The header macroblock number from VP header decoder 52 is 
loaded into macroblock counter 60. Decoding of macroblocks can then resume with 
the updated macroblock count. 

[0089] when no error was detected in the previous video packet, error controller 58 

activates SAD generator 64 to generate the SAD for the macroblocks in the previous 
video packet. The left column and top row of pixels in each macroblock in the 
previous packet are fetched from memory 56 and input to SAD generator 64. These 
pixels can be sent one macroblock at a time, or can be buffered or their SAD's 
generated in parallel. The pixel data from the last row of the macroblocks above are 



[0086] 



Macroblock decoder 54 decodes the macroblocks in the video packet after the VP 
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fetched from last row register 66, while the pixels from the last column of the last 
macroblock are fetched from register 68. Register 68 can be re-loaded with the 
current macroblock's first column of pixels after each current macroblock is processed 
by SAD generator 64. Likewise register 66 can be loaded with the last row of pixels of 
earlier macroblocks. Alternatively, registers 66, 68 can be deleted when memory 56 is 
directly accessed for these pixels. 

[0090] Error controller 58 receives the final SAD from SAD generator 64, and compares 
the SAD to the threshold. When the SAD is above the threshold, an abnormally large 
number of discontinuities have been detected. IUs likely that an error has already 
occurred, but was not detected in the previous packet. The macroblock counter may 
be in error. The header macroblock number from VP header decoder 52 is loaded into 
macroblock counter 60, allowing decoding of macroblocks in the new video packet to 
resume with a correct macroblock number. Error controller 58 can also conceal the 
errors in the previous video packet, such as by using pixels from earlier frames stored 
in memory 56 and discarding the pixels written to memory 56 by pixel writer 62. 
Averaging of pixels from surrounding macroblocks can also be used to conceal the 
error. 

[0091] When the SAD is below the threshold, relatively few discontinuities have been 

detected in the previous video packet. It is likely that the previous video packet was 
correctly processed. It is likely that the macroblock numbers mismatched because the 
header macroblock number was read incorrectly. Thus the header macroblock number 
from VP header decoder 52 is ignored. Instead the macroblock number already in 
macroblock counter 60 is used for identifying macroblocks decoded by macroblock 
decoder 54 in the new video packet. 

[0092] Figure 9 shows border-SAD detection of a bit error in the macroblock data that 
corrupts the macroblock counter. Although start code 20 and VOP header 22 for VOP 
#1 are detected and decoded, a bit error in macroblock data in data field 24 occurs. 

[0093] Some of the early macroblocks MB#1 , MB#2 ... in data field 24 are properly 

decoded, but an error occurs before the last macroblock MB#52 in data field 24 is 
processed. This error is not immediately detected, but the start of some of the 
macroblocks in data field 24 is not recognized. The decoder combines several 
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macroblocks of data together into a single corrupted macroblock. 

[0094] Since some of the macroblocks in data field 24 are not recognized, the 

macroblock counter is not incremented as many times as necessary. The macroblock 
counter is only incremented for recognized macroblocks. In this example, the 
macroblock counter is off by 5, reading 47 for the last macroblock MB#52 in data field 
24. 

[0095] Resync marker 30 for the second video packet (VP) is detected, and VP header 32 
is decoded. A correct value of 53 is read from VP header 32 as the header macroblock 
number. The macroblock counter is pre-incremented from 47 to 48, but does not 
match the header macroblock number of 53. An macroblock-number mismatch error 
is detected. (Step 76 of Fig. 7) However, since the bit error in data field 24 was not 
detected, the SAD for the previous video packet is generated to validate the 
macroblock data in data field 24. 

[0096] The SAD accumulated for macroblocks MB#1 to MB#52 of the previous video 

packet #1 is large, since the macroblock data was corrupted. Once the macroblock 
counter got off count, the remaining macroblocks in the video packet were placed in 
the wrong relative locations in the frame. Many discontinuities were generated along 
macroblock boundaries by the mis-placement of macroblocks. 

[0097] The large SAD is above the threshold, so the error controller assumes that the 
macroblock counter is wrong. The header macroblock number, 53, read from VP #2 
header 32 is loaded into the macroblock counter. The old macroblock counter value of 
48 is over-written. Decoding of macroblocks MB#53 to MB#80 in data field 34 
continue without error, since the correct macroblock number has been loaded into the 
macroblock counter. The macroblock counter is incremented for each macroblock, 
ending video packet #2 with the correct value of 80. 

[0098] Resync marker 31 is detected, and VP #3 header 33 is decoded. The header 

macroblock number of 81 is read from VP header 33, and the macroblock counter 
reads 81 once it is pre-incremented. Since the header and counter macroblock 
numbers match, macroblocks MB#82 to MB#99 are interpreted correctly as 
macroblocks #82 to #99. 
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[0099] The SAD allows the error controller to verify the image decoded for the previous 
video packet. A large SAD value indicates many image edges that coincide with 
macroblock boundaries. The large SAD thus signals a corrupted image, allowing the 
header macroblock number to be used. A small SAD indicates a normal image, 
allowing the macroblock counter to continue to be used. 

[01 00] ALTERNATE EMBODIMENTS 

[01 01] Several other embodiments are contemplated by the inventor. For example the 
decoders, SAD generator, and error controller can be implemented in a variety of 
ways, such as by firmware routines in a digital-signal processor (DSP) chip, or in logic 
in a logic array chip, or as software routines executed by a processor, or a 
combination of techniques. The decoders, controllers, and comparators can be 
partitioned in many different ways. A programmable register can allows SAD 
calculation to be disabled, or allow for different threshold values to be used. 

[0102] Different formulas can be used besides SAD, such as sum of the square of the 

difference. The SAD can be applied to fewer than all macroblocks in the previous video 
packet, or can be generated as the previous macroblock is being decoded rather than 
after an error is detected. The SAD could be used as a general error-detection 
technique, even when macroblock number mismatches are not detected. 

[0103] Other video formats, frame sizes, and macroblock sizes could be supported. Many 
other functional blocks can exist in a complex MPEG decoder, and pipelining logic and, 
staging registers may also be present. 

[01 04] Counters can be reset or initialized to values other than zero, such as to 1 or 

another value. Rather than an increment by 1 , larger adjustments could be made, or 
negative increments or decrements could be used. Various pipelining registers can be 
added. 

[01 05] When the macroblock data is corrupted, the SAD technique may still be useful. For 
example, corrupted macroblocks may have all pixels changed to black or white, 
wiping out any image shapes in the macroblocks. However, discontinuities still can 
occur on macroblock boundaries with non-corrupted macroblocks, or macroblock that 
are corrupted in a different way (different colors, etc.). Rather than obtain the 
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maximum SAD, the SAD's for all macroblocks in a video packet can be summed to get 
an overall SAD for the packet. A running SAD sum can be kept as the macroblocks in 
the video packet are decoded and their SAD's generated. The running sum SAD can be 
normalized by dividing by the number of macroblocks. The SAD can exceed the 
threshold either as a greater than comparison or a greater than or equal comparison. 

[01 06] The abstract of the disclosure is provided to comply with the rules requiring an 
abstract, which will allow a searcher to quickly ascertain the subject matter of the 
technical disclosure of any patent issued from this disclosure. It is submitted with the 
understanding that it will not be used to interpret or limit the scope or meaning of the 
claims. 37 C.F.R. § 1 .72(b). Any advantages and benefits described may not apply to 
all embodiments of the invention. When the word "means" is recited in a claim 
element, Applicant intends for the claim element to fall under 3 5 USC § 1 1 2, 
paragraph 6. Often a label of one or more words precedes the word "means". The 
word or words preceding the word "means" is a label intended to ease referencing of 
claims elements and is not intended to convey a structural limitation. Such means- 
plus-function claims are intended to cover not only the structures described herein 
for performing the function and their structural equivalents, but also equivalent 
structures. For example, although a nail and a screw have different structures, they 
are equivalent structures since they both perform the function of fastening. Claims 
that do not use the word means are not intended to fall under 35 USC § 1 1 2, 
paragraph 6. Signals are typically electronic signals, but may be optical signals such 
as can be carried over a fiber optic line. 

[0107] The foregoing description of the embodiments of the invention has been 

presented for the purposes of illustration and description. It is not intended to be 
exhaustive or to limit the invention to the precise form disclosed. Many modifications 
and variations are possible in light of the above teaching. It is intended that the scope 
of the invention be limited not by this detailed description, but rather by the claims 
appended hereto. 
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